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ABSTRACT 



The University of the Balearic Islands (Spain) has continued 
in recent years to extend and broaden its efforts in the revitalization of 
its undergraduate education program. The University* s location (it covers 
three different islands) makes it a particularly appropriate testbed for 
examining the application of information technologies to undergraduate 
education. This paper describes a work- in-progress project that has involved 
implementing and evaluating a computer-assisted personalized approach for 
homework assignments using the World Wide Web, evaluation tests of students' 
knowledge background, and computer network-based examinations. Students' 
feedback and performance are also discussed. (Author/AEF) 
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Abstract 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



We describe the work-in-progress project of a software tool to implement and evaluate a 
computer-assisted personalized approach for homework assignments using the world 
wide web, evaluation tests of the students’ knowledge background, and examinations, 
currently used in a computer networks course, used for the first time at the University of 
the Balearic Islands. Due to the university’s geographical environment (it covers three 
different islands), we have a specially appropriate testbed in the field of the application of 
information technologies to undergarduate education and, in particular, the computer- 
assisted learning. This short paper describes the first results about the student’s rating of 
the system and performance. 
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The University of the Balearic Islands (UIB) has continued in the recent years to 
extend and broaden its efforts in the revitalization of the undergraduate education. 
The project described in this short paper is one of the last initiatives of a whole 
project, and the work-in-progress is summarized. 

It's well-known that students can learn better when they are stimulated by the 
courseware interactivity and they can follow a personal way to learn. Indeed, some 
courseware prototypes for the specific field of networking learning have been 
reported with high interactivity with good results [9]. Two are the main parts of our 
courseware: 

(A) the course's specific material web pages which includes also URL hnks to 
related items, links to a lot of solved exercises and problems, hnks to homework 
problems, and links to experience questionnaires and knowledge tests, and 

(B) a forum of discussion with advanced features, which virtually gives the 
course as an electronic community. Electronic communities are useful to link 
geographicaUy-dispersed people, which is a potential field of apphcation in our 
islands. Nevertheless, as reported in the hterature, there are key factors for success in 
such electronic groups: (1) that the group have a purpose and specific outcome; (2) 
that at least one person in the group acts as a facihtator; (3) that there is purpose and 
good reason for convening the group electronically. 

Concerning to (A), many web-based courses on networking, with similar 
instructional material and contents, can be found on the web. But what we want to 
emphasize in this paper, and it is a fundamental goal of the project, is our interest to 
study the impact produced on the learning process by the synergy of using the 
potentialities of a forum in conjunction with conventional material. 
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2 General Features of our Approach 



Incorporating software engineering methodologies is a critical aspect [5], [7] 
we are in mind. Also, as computer-science professionals we are leading the project 
because it is difficult for educators that lack technical background to create 
sophisticated www -based courses [6]. In particular, the term computer-mediated 
communications [2] includes, in our context, tools such a web server, video 
conferencing facilities, and a set of wireless area network (WLAN) devices that allow 
to access to teacher's lectures and all educational material at the network-wired web, 
and to the forum, from the classroom, using only a laptop with a wireless device 
attached with a PCMCIA card, and a portable video projector. 

One important limit is the need not to overload the students: questionnaires and 
interviews in order to learn more have the students limited tolerance for this addition 
to their time. So that we have kept a compromise between the amount of feedback 
and the implications of such quizzes on the student performance. The most important 
criterion in testing is whether the courseware is effective in the sense that students 
learns from it. 

About the task experience questionnaire. Targeting particular skills concerning 
the background knowledge of the students, it has been useful for us, the teachers, to 
have some information about the students' previous and current experience in the 
domain of computer networks. The questionnaires have taken the format of an 
anonymous test with about fifty rather-basic questions to be selected among four 
possible answers. The anonymity oh the test was because students were reluctant to 
reveal their identify, worried about the test's implication on its final grade. Anonymity 
was guaranteed because this kind of test did not have any grade's implication and was 
completely different to, and should not be confused with, diagnostic tests to establish 
individual learning needs, that is the knowledge quizzes. 

About knowledge quizzes: the quizzes are constructed similarly to the task 
experience questionnaire. For consistent marking, the quiz is multiple choice: students 
should select the correct answer among (usually) four. Two basic kinds of questions 
were posed which implies: a) the computation of a result and the selection of the 
suitable answer, and b) the selection of a statement on the basis on the truthfulness or 
the falseness suggested by the question. The consistency of a question, in the sense 
that it will be completely understood by the student, has two main implications to be 
carefully taken into account by the teacher: (a) has no ambiguities, i.e.,. only one is 
truth or false, depending on the nature of the question, and (b) in the questions with 
computations, the accuracy of the results has a reasonable degree of tolerance; in 
other words, and the suggested answers are posed on the basis of either with different 
order-of- magnitude or with enough numeric 'distancing'. 

Each question usually corresponds to a distinct learning objective. The 
courseware's teachers interface gives automatically the possibility to enter: i) the 
number of questions of the particular quiz, ii) the text of each question directly or 
from a questions' database, and iii) the number of suggested answers, one and only 
one of each is correct, and the corresponding text. 

3 Feedback for the Instructors 
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The large body of literature that exists concerning the design and evaluation of 
instruction and training systems, and that were though using different distance 
learning technologies [4]. The three issues addressed, which we are taking into 
account in our project to evaluate its effectiveness are: 

(1) Was the course a success in terms of academic performance and proficiency 
gains? 

(2) Was it successful in terms of student satisfaction?, and 

(3) Did the technology support a high level of student interactivity? 

The first issue is a virtual standard for evaluating instructional programs; the 
second issue one may be critical to increase student motivation and to improve its 
learning. Seven parameters are reported in the literature, to be taken into account in 
our context to compute the student satisfaction with the: (a) instructor(s); (b) 
technology (c) course management (d) at-site personnel (d) promptness of material 
delivery relative to timing of the lectures (e) support services; and (f) out-of-class 
interaction with instructor(s). 

The third issue is, namely, relevant to: (i) the real-time availability of the web 
server (ii) the student's accessibility to personal computers at the university and/or at 
home, and (iii) the ergonomics, in the sense of he student's feeling of how easily the 
courseware interface with the course's material is. About (i) the server is has not 
already get a 100% of availability due to maintenance tasks, but it will. As far as 
concerns to (ii), about 180 personal computers are available to students at different 
sites of the campus. Concerning the point (iii), our efforts are, by the one hand, to use 
an interface as ergonomic as possible, interacting with students suggestions and, on 
the other, using an electronic tool for electronic groups. 

As far as we have already get some practical experience, each of these factors 
have a very clear impact in our context. For the forum, the used tool for electronic 
groups is die AltaVista Forum [8] o create a group or "team" of course's members 
(students, teachers, and at-site supporting personnel). This application provides a fun 
and easy way to communicate and to share resources with different groups of people. 
It means that a student can belong to as many teams (courses) as he is registered: with 
the forum, he is viewing his own student profile and enters the specific course by 
clicking on among the list of courses. By now, only the computer networks 
advanced course is on line, but it's likely expected to be other very different courses 
during the next academic year 98-99, as experience is gained with our work. Other 
features make the forum tool very attractive for the computed-assisted learning: every 
course member can communicate with other people, individually or in a broadcast or 
multicast form, by posting notes in an on-line conversation, by requesting an 
appointment with another member of the forum, by having the tool notify members 
through electronic mail when new information has been added to the forum, and by 
having on-line conferences in the form of real-time chatting. In addition to this virtual 
communication channel members can share resources with other by sharing 
documents, by keeping a course calendar of meetings and events, and by searching 
Interment news sources and sharing those with others. This is useful both for teachers 
to get in contact with specific students or to notify the addition of new materials, 
proposals of dates for new quizzes, etc. and for students to get in contact both with 
teachers and other students. The forum tool lets members operate in an environment 
where there is a high level of trust, and because of the number of students of the 
computer networks course is relatively small (about fourteen), that trust is effectively 
possible. Nevertheless, teachers and other support-persons have some privileges: they 



are able to add or delete members, add and delete each other's documents and 
discussion notes . 

4 Student Performance 

In order to analyze the student performance, following partially the method suggested 
in [3], we are observing the following parameters: 

1 . The distribution of the solved homework problems in % for the whole class. 

2. The average performance in the final exam, in %, as a function of the 
fraction of homework problems solved, in %. 

3. The correlation between the percentage of solved homework problems to 
percentage correct on the final exam. 

4. The total number of logins as a function of time during the course. In 
particular to observe its correlation with deadlines of homework, test and exams. 

5. The average number of solved problems, in %, compared with the actual 
degree of difficulty, not the a-priori difficulty estimated by the teachers, defined by 
the ratio of number of wrong answers over the number of total answers, for the 
different homework sets. 

6. The degree of difficulty, expressed as the ratio of number of wrong answers 
over the number of total answers for all of the individual problems (of all homework 
sets). 

5 Concluding Remarks 

The term 'evaluation' implies making a judgment; a better statement is "to discover 
how an educational intervention performs by observing and measuring the teaching 
and learning process, or some small slice of it [1]. By now, as far as the experience is 
still new at our university and the course has not finished, no too much information is 
still available to get reliable conclusions. In the near future, more courses will 
probably adopt our approach and use our experience, so more refined conclusions are 
expected to be available and be reported in future papers. 

Nevertheless, we have observed that one of the main conclusions found in the 
literature [1] is applicable to our experience, which is that the main benefit of 
computer-aided learning is to help teachers make better use of it by adjusting how it 
is used, rather than by changing the software or informing purchasing decisions. 
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